External environment sensitive predictive application and memory initiation

ABSTRACT

An input tracker learns relationships between end user inputs made at a computer and external environment conditions sensed at the computer by external environment sensors integrated in the computer, such as a camera, microphone, hard disk drive motion detector, display ambient light sensor and display orientation accelerometer. An input predictor interfaced with the input tracker applies current external environment conditions sensed by the external environment sensors to the relationships defined by the input tracker to predict future user input and initiates actions by the computer responsive to the predicted future user input before the input is detected to have results responsive to the input prepared for the user upon detection of the input.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates in general to the field of computeroperations, and more particularly to an external environment sensitivepredictive application and memory initiation.

2. Description of the Related Art

Computers save people time. Word processing is an example of this.Documents that used to take teams of secretaries to prepare are nowreadily stored in memory, retrieved and modified as needed by the enduser of the document. Networking is another example of this. Email hasreplaced many mail communications through the post and social networkingwebsites allow nearly instant personal updates to circulate amongfriends. Access to networked information enhances many common tasks,such as by providing employees with ready access to business resourcesand by providing students with ready access to educational materials.End users interact with computers through I/O devices, such as akeyboard, a mouse and a touchscreen. Computers also include a number ofsensors that sense external environment factors for use by componentsand by an end user, such as a camera that captures images, a microphonethat captures audible sounds, a seismic/gyroscopic sensor that capturesmotion to protect a hard disk drive, accelerometers that captureorientation for aligning a display, a GPS sensor that capturespositioning information and an ambient light sensor that senses ambientlight to adjust the brightness of a display screen.

While computers save time in the big picture, often computers seem toeat time when preparing to perform tasks requested by an end user withan I/O device. Consider the employee who sits at his desk at thebeginning of a day. First the employee wiggles the computer mouse orpresses a key at the keyboard to remove the screen saver or wake thecomputer from a sleep state. Next the employee inputs a password andthen requests the email client to display the day's emails. Between eachinteraction, the computer churns information to accomplish the requestedtask by swapping in appropriate bits to display requested information,process inputs, start applications and page requested information tovolatile memory accessible by the processor. Additional delays aresometimes introduced when the computer downloads requested informationfrom a network while the information is communicated from a server.

SUMMARY OF THE INVENTION

Therefore, a computer system and method of use provides more rapidcomputer response times by learning relationships between end userinputs and sensed external environment conditions and then applying therelationships to predict and prepare for end user requests at I/Odevices.

More specifically, an input tracker running on processing resources of acomputer tracks the relationships between end user inputs to thecomputer and external environment conditions sensed by externalenvironment sensors disposed in the computer, such as images captured bya camera, sounds captured by a microphone, motion detected by a harddisk drive motion detector, motion detected by display orientationaccelerometer and other sensors. An input predictor analyzes currentexternal environment conditions sensed by the external environmentsensors and compares the current external environment conditions withstored environment conditions to predict end user inputs at the computersystem. The input predictor responds to a predicted end user input bypreparing actions responsive to the input, such as initiating anapplication and retrieving content from non-volatile storage to volatilestorage for access by a processor. When the predicted input is actuallydetected, the computer responds to the input with improved speed sinceactions responsive to the input have already been performed in responseto the prediction of the input.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be better understood, and its numerousobjects, features and advantages made apparent to those skilled in theart by referencing the accompanying drawings. The use of the samereference number throughout the several figures designates a like orsimilar element.

FIG. 1 depicts a block diagram of a computer configured to predict enduser inputs and to prepare actions responsive to the predicted inputbefore the input is detected;

FIG. 2 depicts a time line of exemplary external environment conditionsand associated actions that are related to predict future inputs at acomputer;

FIG. 3 depicts a flow diagram of a process for learning relationshipsbetween external environment conditions and inputs at a computer; and

FIG. 4 depicts a flow diagram of a process for predicting inputs at acomputer in response to current external environment conditions sensedat the computer.

DETAILED DESCRIPTION

A computer predicts end user inputs so that results are presented forthe inputs in reduced time once the user makes the inputs. In a learningmode, the computer records and stores a moving window or externalenvironment conditions sensed at the computer, such as light, sounds,motions and images sensed during one minute sample periods. The sensedexternal environment conditions are distilled and indexed to generate afingerprint of external conditions during a time period, such as minuteintervals through a work week. The fingerprint may be generated on thefly or on demand when an action occurs for association with theenvironment conditions. When an action is performed on the computer,such as with an input made to an input device of the computer, theaction is encoded and associated with the environment conditionfingerprint sensed at the time of the action. In an act mode, thecomputer records a moving window worth of environment conditions andgenerates a fingerprint for current environment conditions, such aslight, sounds, motions and images sensed at a time of the day and day ofthe week. The computer compares the current fingerprint with the storedfingerprints of external conditions to determine if a match exists. If amatch between current and stored external conditions exists, thecomputer retrieves the action associated with the matching storedexternal conditions and predicts that the action will be initiated atthe computer. In response to the predicted action, the computer preparesfor the action, such as by prefetching memory pages for the applicationor applications associated with the action. Preparing for the input byretrieving stored information and executing instructions reduces thetime need to respond to an end user input for the action when the actionis detected.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon. Anycombination of one or more computer readable medium(s) may be utilized.The computer readable medium may be a computer readable signal medium ora computer readable storage medium. A computer readable storage mediummay be, for example, but not limited to, an electronic, magnetic,optical, electromagnetic, infrared, or semiconductor system, apparatus,or device, or any suitable combination of the foregoing. More specificexamples (a non-exhaustive list) of the computer readable storage mediumwould include the following: an electrical connection having one or morewires, a portable computer diskette, a hard disk, a random access memory(RAM), a read-only memory (ROM), an erasable programmable read-onlymemory (EPROM or Flash memory), an optical fiber, a portable compactdisc read-only memory (CD-ROM), an optical storage device, a magneticstorage device, or any suitable combination of the foregoing. In thecontext of this document, a computer readable storage medium may be anytangible medium that can contain, or store a program for use by or inconnection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device. Program codeembodied on a computer readable medium may be transmitted using anyappropriate medium, including but not limited to wireless, wireline,optical fiber cable, RF, etc., or any suitable combination of theforegoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks. These computerprogram instructions may also be stored in a computer readable mediumthat can direct a computer, other programmable data processingapparatus, or other devices to function in a particular manner, suchthat the instructions stored in the computer readable medium produce anarticle of manufacture including instructions which implement thefunction/act specified in the flowchart and/or block diagram block orblocks. The computer program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other devicesto cause a series of operational steps to be performed on the computer,other programmable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

Referring now to FIG. 1, a block diagram depicts a computer 10configured to predict end user inputs and to prepare actions responsiveto a predicted input before the input is detected at an input device.Computer 10 processes information by executing instructions with aprocessor 12. Instructions and other information that support processoroperations are stored in volatile memory, such as dynamic random accessmemory (RAM) 14. A hard disk drive 16 or other non-volatile memorystores information during powered down states so that processor 12 canretrieve the information to RAM 14 for execution when processor 12powers up. During normal operations, an operating system 18 manages theexecution of applications 20 to perform tasks commanded by an end userthrough input devices, such as a keyboard 22 and mouse 24. The resultsof the commanded tasks are presented at a display 26, which may alsoinclude a touchscreen input device 28. In some instances, an application20 performs communication with an external network through a networkinterface card (NIC) 30, such as an Ethernet or wireless networkcommunication interface. Examples of applications 20 that use networkcommunications include web browsing and e-mail.

Firmware 32 coordinates on a physical level the interactions betweenprocessor 12, RAM 14, hard disk drive 16, keyboard 22, mouse 24, display26 and NIC 30. Firmware 32 is, for example, a chipset having an embeddedcontroller, keyboard controller and/or other processing resources thatexecute instructions stored in non-volatile flash memory. Firmware 32also coordinates interfaces between processor 12 and a variety ofexternal environment sensors so that external environment conditions maybe used by applications 20 running on processor 12. For example, acamera 34 captures images and a microphone 36 captures sounds for use bya video conferencing application. Other types of external environmentsensors that sense external environment conditions at computer 10include: a Global Positioning System (GPS) 38 that detects currentposition, a temperature sensor 39 that senses thermal conditions, anorientation accelerometer 40 that detects orientation of display 26 todisplay information upright relative to gravity, an ambient light sensor42 that detects ambient light to adjust the brightness of display 26,and a motion detector 44 integrated in hard disk drive 16 to protecthard disk drive 16 from impact shock damage, such as aseismic/gyroscopic detector.

In order to minimize delays associated with user inputs made at inputdevices keyboard 22 and mouse 24, an input predictor 46 monitorsexternal environmental conditions to predict future inputs and initiatesactions at computer 10 that improve the response time of computer 10 ifthe predicted input is actually made. For example, delay between a userinput and a computer response may be introduced by retrieval ofinformation from non-volatile memory of hard disk drive 16 to volatilememory of RAM 14 for use by processor 12, such as retrieval ofinstructions to execute an application 20 or retrieval of content forpresentation by an application 20. Input predictor 46 reduces suchdelays by initiating applications and/or retrieval of informationresponsive to a predicted input before the predicted input is made by anend user. Input predictor 46 predicts future end user inputs bymonitoring external environment conditions detected by externalenvironment sensors, such as camera 34, microphone 36, GPS 38,temperature sensor 39, accelerometer 40, ambient light sensor 42 andmotion detector 44. If external environment conditions are detected thatindicate a predetermined end user input will be initiated, then inputpredictor 46 predicts the input and initiates actions to reduce delayshould the predicted input be detected at an input device.

Input predictor 46 bases predictions of future inputs on historical datagathered by an input tracker 48 during monitoring of normal use ofcomputer 10 by an end user. For example, input tracker 48 samplesexternal environment conditions detected by the external environmentsensors over defined time periods. In one example embodiment, inputtracker 48 references clock 50 to sample external environment conditionsin one minute samples over the course of a normal work week to identifypatterns of external conditions associated with inputs made to inputdevices. The samples and/or the patterns are stored in aninput/environment relationship table 52 that is accessible by inputpredictor 46. Input predictor 46 accesses input/environment relationshiptable 52 to compare currently detected external environment conditionswith historical patterns and, if current conditions match a historicalpattern, then input predictor 46 predicts an input associated with thematching historical pattern. Based upon the predicted future input,input predictor 46 initiates actions at computer 10 to prepare for theinput, such as retrieving and activating one or more applications andretrieving content to prepare the content for presentation at display26. In one embodiment, input predictor prepares for the future input butwithholds presentation of information for the predicted input until theactual input is received. Withholding output generated based upon apredicted input reduces end user annoyance at unwanted automated actionsand provides a near instantaneous response when the predicted input ismade. In one embodiment, a user interface 53 interacts withinput/environment relationship table 52 so that an end user can setfingerprints of predicted actions based upon external environmentconditions or alter predictions made by input predictor 46.

Input predictor 46, input tracker 48 and input/environment table 52 maybe performed by an application running on processor 12, by instructionsrunning as firmware 32 or a combination of processing resources. In oneexample embodiment, input predictor 46 runs as firmware 32 whileprocessor 12 is powered down, such as in a sleep or hibernate state.Input predictor 46 provides power to one or more of the externalenvironment sensors to detect external environment conditions predictiveof initiation of an on state from the sleep or hibernate state. Forinstance, an increase in light that indicates a user has entered theroom or the sound of keys opening a door at the start of a business daymatch external conditions associated with a power up input and requestfor email. In response to such external conditions, input predictor 46running in firmware 32 initiates a power up of computer 10 and calls theemail application to present email at display 26. During the sleep orhibernate state, input tracker 48 runs in firmware 32 to track sensedexternal environment conditions and store the conditions ininput/environment relationships table 52. Once computer 10 enters the onstate, input predictor 46 initiates as an application running onprocessor 12 to analyze the sensed environment conditions stored duringthe powered down state. In one embodiment, input predictor 46 looks forpatterns that associate external environment conditions with inputs byrunning a neural network or other predictive algorithms during idle timeof processor 12. In one alternative embodiment, input predictor 46 actsto prevent automated actions by computer 10 where the automated actionswould be countered by a predicted input. For example, input predictor 46counters an automated transition to a powered down state if camera 34detects a user image and detection of the image indicates the user wouldmake an input in response to the powered down state that would counterthe powered down state, such as a mouse or keyboard input to bringcomputer 10 back to an on state.

During normal operations, input predictor 46 samples externalenvironment conditions sensed by camera 34, microphone 36, GPS 38,accelerometer 40, ambient light sensor 42 and motion detector 44 todetect external environment conditions that match those associated withan end user input stored in input/environment relationship table 52.Input tracker 48 learns external conditions associated with end userinputs by storing a moving window of sensed external environmentconditions and associating user input device inputs with the sensedexternal environment conditions. For example, minute long samples ofexternal environment conditions are saved when an input is made by anend user and associated with the input. Input predictor 46 monitorsexternal environment conditions in a similar moving window, such asminute long increments, and compares the currently sensed externalenvironment conditions with historical conditions to identify a match.If a match exists between current and historical external environmentconditions, input predictor 46 predicts the input associated with thematched external environment conditions and initiates actions atcomputer 10 to meet the requirements of the predicted input. In oneembodiment, input predictor saves information at an address ofnon-volatile storage hard disk drive 16 to retrieve information for apredicted input, such as a script of one or more applications to run,content to retrieve, or executable code retrievable for execution byprocessor 12 that will result in the state associated with a predictedinput being achieved at computer 10 without having to execute allinstructions normally required by the input.

Referring now to FIG. 2, a time line depicts exemplary externalenvironment conditions and associated actions that are related topredict future inputs at a computer. At 4 A.M., a janitor cleaning anoffice where computer 10 is located turns on a light that is detected bycamera 34 and ambient light sensor 42, and runs a vacuum that generatesnoise detected by microphone 36. Since no inputs are detected at inputdevices of computer 10, no actions are associated with the janitor'sexternal environment conditions. In one embodiment, the janitor'sconditions are too remote to any inputs and are therefore not tracked ininput/environment relationship table 52.

At 8:30 A.M., the employee assigned to the office arrives at work, opensthe door with his keys, turns on the lights, approaches the computer andmoves the mouse to bring up a login page. Next the employee inputs hispassword and opens an email application to check email, a web browser tocheck news and a music application to play Chopin's Fantasy Impromptu.Input tracker 48 detects the key sound with microphone 36, the lightwith camera 34, and an image of the end user with camera 34, andassociates these detected environmental conditions during a time framearound 8:30 A.M. with the inputs for bringing up the login page andretrieving email, news and music. The relationship between the sensedenvironment conditions and the inputs are stored in input/environmentrelationship table 52 and made available to input predictor 46 for usethe next morning. On the next morning, when input detector 46 detectsthe sound of keys and the lights at around 8:30 A.M., it predicts alogin and retrieval of email, web browsing news and music. In responseto the predicted inputs, input predictor 46 immediately initiatespresentation of the login page as if the end user had already touchedthe mouse, and initiates the email, web browsing and music activities.By the time the end user has completed login, email, web browsing andmusic are loaded into volatile memory so that each input by the end useris met with near instantaneous response.

At 10:30 A.M., the employee picks up the computer and walks to ameeting. At the meeting, the employee opens a word processing documentcalled “meeting notes.” Input tracker 48 senses accelerations associatedwith walking with computer 10 by display orientation accelerometer 40and hard disk drive motion detector 44, as well as lighting changes thatoccur during the walk, such as a walk across a courtyard in daylight.The position of computer 10 is sensed as changing by GPS 38. Inputtracker 48 associates the sensed external environment conditions at10:30 A.M. with the word processing document used at the meeting anddefines the relationship in the input/environment relationships table52. The next day at around 10:30 A.M., input predictor 46 senses themotion, light and GPS environment conditions with the externalenvironment sensors and finds the match of external environmentconditions in input/environment relationship table 52. Based on thematch between current external environment conditions and the historicalconditions stored in table 52, input predictor 46 predicts the input tostart the word processing program and to retrieve the “meeting notes”content. In response to the predicted inputs, input predictor 46initiates the word processing application and retrieves “meeting notes”from hard disk drive 16 to RAM 14, but withholds presentation of theword processing application and “meeting notes” document until the enduser makes the predicted inputs. In alternative embodiments, alternativeinputs may be predicted based upon alternative external environmentconditions and the time frame in which the conditions occur. End usersmay manually set conditions that will predict actions or may manuallyremove predicted actions from input/environment relationship table 52.

Referring now to FIG. 3, a flow diagram depicts a process for learningrelationships between external environment conditions and inputs at acomputer. At step 54, the computer stores a moving window, such as oneminute increments, worth of external environment conditions detectedwith sensors of the computer, such light, audio and motion conditions.At step 56, the external environment conditions are processed to distillidentifiable features of each condition and indexed to generate afingerprint. The fingerprint is generated on the fly or on demand wheninputs are detected at input devices of the computer when a relationshipis defined that associates external environment conditions with aninput. The type of information distilled from the external environmentsensors can include general values, such as light intensity or noiseloudness, or more specific values, such as a facial image or voice tone.At step 58, when an action is performed at the computer by an end userinput, the action is encoded and associated with the fingerprintdetected at the time of the action. The process then returns to step 54to continue learning about the environment. During idle times at thecomputer, analysis may be performed by the computer processor on thesaved fingerprints to identify patterns of external environmentconditions and inputs where predictive responses will improve computerresponse time.

Referring now to FIG. 4, a flow diagram depicts a process for predictinginputs at a computer in response to current external environmentconditions sensed at the computer. At step 60, the computer records amoving window of sensed external environment conditions in a manner thatwill allow a comparison with historical external environment conditions,such as using the same one minute increments. At step 62, a fingerprintis generated that will allow a comparison with historical fingerprintsof external environment conditions. At step 64, the currently sensedfingerprint is looked up in the index of historical fingerprints todetermine if a match exists between current conditions and a historicalcondition. If no match is determined at step 64, the process continuesto step 66 to discard the currently sensed conditions and then returnsto step 60 to record the next moving window. If a match exists at step64, the process continues to step 68, the actions associated with thehistorical fingerprint are predicted for the currently sensed conditionsand the computer system responds by initiating the predicted inputs forthe actions before the inputs are detected, such as by prefetchingmemory pages for the applications used in a tagged event.

Using the timeline of FIG. 2 as an example, an historical fingerprint isestablished that between 8:00 and 9:00 A.M. the microphone detects thesound of a key, the sound of a door opening and the sound of a lightswitch, and the camera detects a light turned on. This historicalfingerprint is associated with the actions of logging in at the computerand swapping an email client, browser and music player to theforeground. Therefore, when current external environment conditions havea fingerprint that matches the stored fingerprint, the computer respondsby prefetching the login panel and authentication application and bypaging the email client, web browser and music player to main memory,even though the end user has not yet directly interacted with thecomputer input devices. As a result, by the time the end user sits atthe computer and selects the predicted inputs, the login, email, browserand music respond promptly. In addition to performing innocuous,under-the-cover operations like memory paging, more foreground-orientedactions may be performed, such as initiating an predicted applicationthat is not already running, triggering replication and transitioningthe computer from a power down state, such as sleep and hibernatestates, to an on state. Default actions may be configured by an end userso the computer responds in a desired manner to a predetermined externalenvironment condition.

Although the present invention has been described in detail, it shouldbe understood that various changes, substitutions and alterations can bemade hereto without departing from the spirit and scope of the inventionas defined by the appended claims.

What is claimed is:
 1. A computer system comprising: a processoroperable to execute instructions; volatile memory interfaced with theprocessor and operable to store the instructions in support of executionof the instructions by the processor; non-volatile memory interfacedwith the processor and operable to store the instructions in a powereddown state; a display interfaced with the processor and operable topresent content generated by the instructions; one or more input devicesinterfaced with the processor and operable to accept inputs from an enduser; firmware interfaced with the processor and operable to coordinateinteraction between the processor, volatile memory, non-volatile memory,display, and one or more input devices; one or more external environmentsensors interfaced with the firmware, each external environment sensoroperable to sense an external environment condition; and an inputpredictor interfaced with the one or more external environment sensorsand operable to predict a future input to the one or more input devicesbased upon one or more external environment conditions sensed by the oneor more external environment sensors, the future input comprising atleast initiation of an application, the input predictor further operableto load instructions for executing the application from non-volatilememory to volatile memory for execution by the processor in response tothe predicted input and to execute the instructions to initiate theapplication in response to prediction of the future input, wherein theinput predictor comprises instructions running in the firmware and theexternal environment sensor comprises a microphone, the input predictorpredicting an input to request power up of the processor from a sleepstate to an on state based at least in part upon a sound captured by themicrophone of keys unlocking a door.
 2. The system of claim 1 whereinthe external environment condition sensed by the external environmentsensors comprises the sound of keys unlocking a door and the inputs madeby a user at the input device comprise power up of the processor andexecution of a predetermined application.
 3. A system for prefetchingcontent for presentation at a computer, the system comprising: aprocessor to execute instructions; memory storing the instructions, theinstructions including: an input tracker interfaced with plural externalenvironment sensors and at least one input device, the input trackeroperable to relate external environment conditions sensed by theexternal environment sensors in a relationship with inputs made by auser at the input device; and an input predictor interfaced with theinput tracker, the input predictor operable to apply current externalenvironment conditions sensed by the external environment sensors to therelationship of the input tracker to predict a future user input at theinput device, and in response to the predicted future user input toexecute an application associated with the future user input on theprocessor, to prefetch content associated with the application, toprepare the content for presentation as visual images at a display, andto withhold presentation of the visual images until detection of thefuture user input, wherein the external environment condition sensed bythe external environment sensors comprises the sound of keys unlocking adoor and the inputs made by a user at the input device comprise power upof the processor and execution of a predetermined application.
 4. Thesystem of claim 3 wherein the input predictor is further operable topower up the processor in response to detection of the sound of keysunlocking a door.